Fix OpenAI streaming tool call chunk merging#6525
Closed
liushenling2001 wants to merge 2 commits into
Closed
Conversation
OpenAI-compatible providers such as vLLM can stream a single tool call across multiple chunks, with id, function name, and function arguments arriving separately. Some providers also repeat the same tool call id on later chunks. The previous merger appended chunks whenever an id was present, which could split one logical tool call into multiple incomplete calls. That led to stream aggregation failures and later tool execution errors such as missing or empty tool names. Merge tool call chunks by index/id, preserve the first function name, concatenate argument fragments, and add regression coverage for split fields and repeated ids.
Member
|
Hi @liushenling2001, is this a follow up to #6441 (comment)? |
Contributor
|
For review context, this overlaps with #6381 in the #6504 already covered the empty |
Member
|
Hi @liushenling, I just merged #6381 which fixes the same issue with a slightly more readable codeflow IMO. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix OpenAI streaming tool call chunk merging for OpenAI-compatible providers.
Some OpenAI-compatible servers, observed with vLLM-backed endpoints, stream one logical tool call across multiple chunks. The tool call id, function name, and function arguments can arrive in separate chunks, and some providers repeat the same tool call id on later chunks.
The previous
ChunkMergerappended every chunk that contained a tool call id as a new tool call. That could split one logical tool call into multiple incomplete tool calls. In practice this caused stream aggregation or downstream tool execution failures, including errors like:NoSuchElementException/ aggregation errors while converting merged chunkstoolName cannot be null or emptywhen an incomplete merged tool call reached tool executionNo ToolCallback foundwhen the logical function metadata was split incorrectly across chunksChanges
indexor repeatedidinstead of appending every id-bearing chunk.id,function.name, andfunction.argumentsarriving in separate chunksVerification
Ran:
mvn -pl models/spring-ai-openai -Dtest=OpenAiChatModelTests testResult:
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0.